Skip to content

feat(document): add bounded CSV to PDF conversion - #293

Merged
mariusvniekerk merged 2 commits into
kenn-io:mainfrom
rodboev:pr/175-2-mistral-csv
Sep 6, 2026
Merged

feat(document): add bounded CSV to PDF conversion#293
mariusvniekerk merged 2 commits into
kenn-io:mainfrom
rodboev:pr/175-2-mistral-csv

Conversation

@rodboev

@rodboev rodboev commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Embedded applications can convert CSV locally into a bounded PDF for the existing OCR path, with hashes for both files and a mapping from generated pages to CSV records and cells. The converter checks source, cell, page and output limits before returning a PDF, so callers can replace their own PDF conversion step with a checked Go API.

This adds the CSV conversion library and its documentation. Provider authorization and application integration remain separate, including the Msgvault work in kenn-io/msgvault#608.

Refs #175.

@roborev-ci

roborev-ci Bot commented Sep 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (4a1f2b3)

Verdict: Two medium-severity issues require attention before merging.

Medium

  • document/mistral/capabilities.go:63-65, 193-194 — Adding CSV as an expected bound can invalidate previously valid schema-3 manifests, preventing authorization even for PDF uploads. Preserve legacy unbound CSV results as non-authorizing or add manifest-version compatibility/migration handling.

  • document/mistral/units.go:14, document/mistral/spool.go:298-303Prepare records CSV counts above MaxUnits without rejecting them, causing processing to report ErrorCapabilityChanged instead of a normal input-limit error. Reject over-limit counts during preparation or classify the validation failure separately.


Reviewers: 2 done | Synthesis: codex, 14s | Total: 16m28s

@roborev-ci

roborev-ci Bot commented Sep 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (a552307)

Summary: Medium-severity issue identified in CSV unit-bound enforcement.

  • Medium — CSV local-exact bound can be exceeded after egress

    CountCSVRecords counts each CSV row as one unit, but a large or multiline quoted row may render as multiple provider pages. Since the provider-side mismatch check occurs only after the full source is sent, a valid CSV can exceed MaxUnits after egress.

    References: document/mistral/units.go:14, document/mistral/rendition.go:164


Reviewers: 2 done | Synthesis: codex, 7s | Total: 13m57s

@rodboev
rodboev marked this pull request as draft September 6, 2026 01:54
@rodboev rodboev changed the title feat(rendition): authorize CSV Mistral OCR inputs feat(document): add bounded CSV to PDF conversion Sep 6, 2026
@rodboev
rodboev force-pushed the pr/175-2-mistral-csv branch from a552307 to d511afd Compare September 6, 2026 03:08
@roborev-ci

roborev-ci Bot commented Sep 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (d511afd)

No issues found.


Reviewers: 2 done | Synthesis: codex | Total: 3m33s

@rodboev
rodboev marked this pull request as ready for review September 6, 2026 03:16
@mariusvniekerk
mariusvniekerk merged commit 5969c0b into kenn-io:main Sep 6, 2026
14 checks passed
wesm pushed a commit to kenn-io/msgvault that referenced this pull request Sep 7, 2026
…rsion (#791)

Standalone CSV attachments become searchable through the existing document search commands, including semantic and hybrid retrieval when document embeddings are enabled. The worker converts each opted-in CSV locally with Docbank's bounded CSV to PDF converter, then sends only the generated PDF through the existing authenticated PDF capability and consent.

The archive keeps the CSV attachment's source hash and media type. It records the generated PDF hash, size, page count, converter identity, policy fingerprint and page to record and cell spans as extraction provenance. The generated PDF remains transient. Direct capability-authorized formats continue through their existing routes and upload their original bytes. Raw CSV has no enforceable Mistral unit bound, so disabled conversion leaves it outside the authorized scope. Each extraction records the media type it uploaded through. A served head covers its owner only while the in-scope representative occurrence still selects that route, so the candidate scan reprocesses a hash whose route changed through an occurrence, message type, or liveness change.

CSV conversion is off by default under `[attachments.documents.conversion.csv]`. Enabling it or changing an effective source, response, or page bound creates a new exact extraction profile and requires `msgvault documents consent-mistral` again. Existing PDF-only profiles keep their current identity. The dependency update to kenn-io/docbank#293 also preserves existing document-vector fingerprints through a small compatibility port for the removed raw embedding API.

The Docbank dependency also raises the first-party `go.kenn.io/kit` module from v0.18.1 to v0.24.1 through minimal-version selection. The transitive `golang.org/x/image` module is pinned to v0.45.0 to clear the hosted vulnerability check.

Refs #608



Co-authored-by: Rod Boev <rodboev@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants